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Executive  Summary 


Phase  II  of  the  tactical  airspace  integration  system  (TAIS)  situation  awareness  (SA)  integration 
into  the  cockpit  crew  station  working  group  (CSWG)  was  conducted  19-20  January  2010  in 
support  of  the  product  manager,  air  traffic  control  (PM-ATC).  The  purpose  of  the  CSWG  was  to 
allow  pilots  (users),  designers,  human  factors  engineers,  and  representatives  of  PM-ATC  to 
evaluate  and  participate  in  the  CSWG.  Results  indicated  that  with  the  integration  of  information 
pertaining  to  dynamic  airspace  updates,  the  overall  air  crew  (particularly  the  copilot)  experienced 
a  reduction  in  workload,  a  decrease  in  visual  workload,  and  an  increase  in  SA.  Consequently, 
these  data  indicated  that,  within  increasingly  congested  airspace,  integrating  dynamic  airspace 
information  into  the  cockpit  should  be  minimized  to  reduce  the  risk  of  fratricide  and  controlled 
flight  into  terrain.  Four  experienced  U.S.  Army  rotary- wing  pilots  from  a  government  and 
contractor  consortium  were  selected  as  the  user  group.  A  series  of  three  vignettes  were  flown  in 
a  generic  cockpit  representing  analog  capabilities  with  the  support  of  an  electronic  data  manager 
(EDM).  The  pilots  were  briefed  to  fly  an  air  movement  of  troops  from  the  pickup  zone  to  the 
landing  zone  and  return  to  the  assembly  area.  Throughout  the  mission,  dynamic  airspace  events 
were  introduced  into  the  cockpit.  The  first  vignette  represented  current  capabilities,  in  which 
only  preloaded  flight  information  was  presented  on  the  EDM  while  dynamic  airspace  events 
were  communicated  via  radio.  In  the  second  vignette,  dynamic  airspace  information  was 
automatically  transmitted  to  the  EDM  from  the  TAIS.  In  the  third  vignette,  dynamic  airspace 
information  from  TAIS  was  presented  with  an  auditory  annunciation. 


Intentionally  lelt  blank. 


1.  Introduction 


1.1  Background  and  Purpose 

The  tactical  airspace  integration  system  (TAIS)  situational  awareness  (SA)  integration  into  the 
cockpit  effort  consists  of  a  series  of  Crew  Station  Working  Group  (CSWG)  meetings  that 
evaluate  the  effects  of  dynamically  integrating  TAIS  symbology  into  the  cockpit.  This  will 
provide  pilots  with  near-real-time  airspace  information  in  an  increasingly  dynamic  and  often 
congested  airspace  environment.  This  specific  study  aimed  to  measure  air  crew  workload 
(WLD)  and  SA  variations  resulting  from  the  introduction  of  TAIS  symbology  into  currently 
fielded  analog  aircraft  in  which  electronic  data  managers  (EDMs)  are  used  for  navigation. 

The  U.S.  Army  Research  Laboratory  (ARL)  Human  Research  and  Engineering  Directorate 
(HRED)  conducted  the  assessment  of  WLD  and  SA  that  pilots  experienced  while  flying 
simulated  missions.  The  assessment  was  conducted  to  determine  the  effects  of  integrating 
dynamic  airspace  symbology  from  TAIS  automatically  into  the  copilot’s  electronic  data  manager 
(EDM)  as  compared  to  similar  information  verbally  relayed  to  the  pilots  by  an  air  traffic 
controller  (ATC).  The  cognitive  decision  aiding  system  had  to  provide  a  workload  environment 
for  the  crew  with  a  Bedford  workload  rating  scale  (BWRS)  mean  score  not  exceeding  6.0. 

1.2  Simulation  Description 

A  simulated  analog  cockpit  was  used  to  address  the  near-term  potential  goal  of  integrating  TAIS 
symbology  into  cockpits  in  which  EDMs  are  commonly  used.  The  EDM  was  linked  to  a  TAIS 
box  via  Winzel,  which  allowed  dynamic  airspace  control  measures  (ACMs)  and  air  tracks  to  be 
transmitted  automatically  to  the  EDM.  Flight  scenarios  were  flown  in  the  battlefield  highly 
immersive  virtual  environment  (BHIVE)  located  in  the  System  Simulation  Development 
Directorate  (SSDD)  19-20  February  2010.  There  were  two  multifunctional  displays  (MFDs)  for 
each  pilot  but  only  one  was  enabled,  which  presented  information  similar  to  analog  cockpits  for 
each  pilot.  For  navigation  information,  the  copilot  relied  primarily  on  the  outside  the  window 
(OTW)  scene  and  the  digital  map  presented  on  the  EDM. 

1.3  Assessment  Overview 

The  assessment  consisted  of  operational  missions  conducted  by  experienced  pilots  in  the 
BHIVE.  Simulation  training  was  not  required  because  all  pilots  were  experienced  with  the 
particular  cockpit  and  simulation  environment.  Before  entering  the  simulation,  pilots  were 
provided  a  mission  brief  and  their  flight  plan.  Prior  to  the  beginning  of  data  collection,  the 
copilot  was  fitted  with  the  head  and  eye  tracker,  which  was  then  calibrated.  The  mission 
scenario  was  a  standard  air  escort  mission  developed  by  subject  matter  experts  (SMEs)  within 
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SSDD  in  accordance  with  established  aircraft  tactics,  techniques,  and  procedures.  The  pilots 
flew  the  same  mission  for  vignette  1  (VI),  vignette  2  (V2),  and  vignette  3  (V3),  but  different 
capabilities  were  introduced  during  each  vignette. 

Each  mission  lasted  roughly  40  minutes,  after  which  pilots  were  sequestered  to  a  calm 
environment  where  air  crew  responses  were  captured  with  surveys  and  the  after  action  review 
(AAR)  process. 


2.  Method 


2.1  Data  Collection 

Pilot  workload,  SA,  visual  dwell  times,  audio- video  recordings,  flight  performance,  and 
open-ended  survey  data  were  collected  and  analyzed.  These  data  were  assessed  to  determine: 

(1)  the  SA  effect  of  bringing  TAIS  symbology  into  the  cockpit  in  real  time,  (2)  the  workload 
implications  for  accessing  dynamic  airspace  update  (DAU)  information  with  and  without  TAIS 
SA  integration  support,  (3)  flight  performance  of  the  pilot  crew  in  terms  of  efficient  deviations 
from  restricted  operating  zones  (ROZs)  that  obstruct  the  flight  path,  and  (4)  pilot/user 
community  attitudes  and  feedback  regarding  the  capability. 

The  BWRS,  SA  reporting  technique  (SART),  and  open-ended  questionnaires  were  developed  in 
accordance  with  published  guidelines  for  proper  format  and  content  (O’Brien  and  Charlton, 
1996).  After  each  flight  scenario,  the  pilot  and  copilot  completed  the  BWS,  SART,  and  an 
open-ended  questionnaire.  For  the  purpose  of  this  study,  each  questionnaire  packet  was 
modified  for  relevant  pilot  and  copilot  tasks  related  to  the  use  of  the  EDM  with  and  without 
TAIS  SA  integration.  Additional  data  were  collected  from  the  pilots  and  SMEs  via  real-time 
observation  of  the  flight  scenarios,  postmission  discussions,  and  AARs. 

2.2  Evaluation  Design 

The  nature  of  this  evaluation  focused  on  operational  realism,  which  was  most  appropriate  for 
fulfilling  customer  requirements,  rather  than  internal  validity.  However,  multiple  variables  were 
controlled  to  maximize  the  validity  and  ensure  that  the  results  can  be  generalized.  To  evaluate 
the  effects  of  TAIS  SA  being  transmitted  to  the  EDM,  pilot  and  crew  performance  was  captured 
under  three  conditions:  (1)  current  capability,  (2)  TAIS  SA  integration  into  the  cockpit,  and  (3) 
TAIS  SA  integration  into  the  cockpit  with  annunciations  (alerts  that  a  dynamic  event  has  been 
transmitted  to  the  EDM).  Pilot  experience  (expert  level),  flight  scenario,  training,  lighting 
conditions,  and  seat  position  were  held  constant  to  control  for  potential  confounds.  The  right 
seat  was  designated  as  the  pilot  and  left  seat  as  the  copilot,  with  the  EDM  user  always  in  the  left 
seat.  Each  crewmember  flew  at  least  one  of  three  flight  scenarios  in  the  copilot  seat  to  collect 
EDM  user  data  from  all  pilot  perspectives. 
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2.3  Flight  Vignettes 


Three  different  flight  vignettes  were  flown  by  each  pilot  crew,  representing  three  variations  of 
SA  information  transmission  strategies.  Each  crew  remained  constant  throughout  the  vignettes 
Ten  DAU  items  were  introduced  into  the  cockpit  for  each  vignette  (see  table  1).  The  purpose 
was  to  evaluate  the  effects  of  transmitting  DAU  to  the  pilot  crew  in  three  different  ways.  The 
first  vignette  simulated  the  current  capability  of  transmitting  DAUs  into  the  cockpit.  In  this 
condition,  pilots  loaded  the  aircraft  with  their  aviation  mission  planning  system  (AMPS)  card, 
consisting  of  three  preloaded  items  represented  by  the  gray  shaded  boxes  in  table  1.  The 
remaining  seven  items  were  introduced  to  the  pilots  after  they  took  off  and  reached  predefined 
locations  in  which  each  DAU  was  presented.  The  flight  route  is  shown  in  figure  1. 

Table  1.  Air  control  measure  items. 


1. 

Control  zone  at  Pickup  Zone  Irwin 

2. 

ROZ  “Al”  displayed  between  SP1  and  ACPI 

3. 

Raven  unmanned  aerial  vehicle  (UAV)  air  track  after  ACP2 

4. 

ROZ  “C3”  displayed  between  ACP4  and  ACP5 

5. 

ROZ  “F5”  at  Landing  Zone  Devil 

6. 

Air  corridor  “Blue” 

7. 

Predator  air  track  in  air  corridor  “Blue” 

8. 

Air  corridor  “Chevy” 

9. 

ROZ  “A3”  at  “Dog  Leg”  on  air  corridor  “Chevy” 

10. 

Flight  of  two  (2)  AH-64  air  track  in  air  corridor  “Chevy” 

Figure  1.  Preplanned  flight  route. 
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During  the  first  vignette,  each  DAU  was  transmitted  via  voice  communication  from  the  battle 
master  station  to  the  simulated  cockpit.  Here,  pilots  had  to  write  down  altitude  and 
latitude/longitude  coordinates  communicated  by  the  ATC.  After  that,  the  copilot  compared  the 
coordinates  to  coordinate  locations  on  the  EDM  digital  map  to  find  physical  landmarks  that 
he/she  could  then  reference,  and  then  recommended  a  plan  of  action  to  the  pilot. 

The  second  vignette  consisted  of  the  same  ACMs  and  flight  route.  However,  when  the  DAUs 
were  introduced  into  the  flight  scenario,  they  were  transmitted  directly  onto  the  EDM  digital 
map,  allowing  the  copilot  to  acquire  the  spatial  location  of  potentially  threatening  airspace 
objects  by  simply  viewing  their  location  on  the  digital  map.  Current  capability  would  dictate  that 
this  information,  if  received,  would  be  “mentally  calculated”  and  visualized  by  the  copilot  to 
locate  the  ACM  on  the  digital  map  based  on  coordinates  received  by  the  ATC. 

The  third  vignette  consisted  of  the  same  ACM  items  and  flight  route,  but  introduced  verbal 
annunciations  from  the  ATC  when  the  DAU  was  introduced.  The  purpose  of  this  was  to 
evaluate  how  an  annunciation  of  new  ACM  information  affected  WLD  and  SA.  The  expectation 
was  that  a  verbal  annunciation  would  alleviate  visual  resources  used  to  monitor  the  airspace 
instead  of  the  digital  map  display,  facilitating  a  more  beneficial  allocation  of  attention. 

2.4  Participants  and  Demographics 

Four  U.S.  Army  rotary- wing  aviators  participated  in  this  event.  The  demographics  questionnaire 
consisted  of  basic  information  on  each  pilot.  Flying  experience  for  each  pilot  across  major 
aircraft  platforms  is  presented  in  table  2. 


Table  2.  Pilot  flying  experience. 


Pilot  PIN 

Qualified  Platform 

IP  Hours 

PI  Hours 

PIC  Hours 

NVD  Hours 

001 

OH-58A 

— 

200 

— 

— 

OH-58D 

— 

450 

— 

— 

UH-1 

— 

150 

— 

— 

UH-60 

1500 

750 

2500 

750 

002 

UH-1 

— 

228 

100 

50 

CH-47 

2500 

750 

750 

1700 

003 

OH-58A 

1000 

500 

500 

600 

UH-1 

— 

125 

— 

50 

UH-60 

1500 

1000 

1500 

1000 

004 

UH-1 

1000 

200 

1500 

150 

CH-47 

800 

100 

1500 

150 

Totals 

OH-58A 

1000 

700 

500 

600 

OH-58D 

— 

450 

— 

— 

UH-1 

1000 

703 

1600 

250 

UH-60 

3000 

1750 

4000 

1750 

CH-47 

3300 

850 

2250 

1850 

Overall 

— 

8300 

4453 

8350 

4450 

Total  flight  time  across  all  platforms  =  25,553 


Note:  PIN  =  personal  identification  number;  IP  =  instructor  pilot;  PI  =  pilot;  PIC  =  pilot  in  command;  and 
NVD  =  night  vision  device. 
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2.5  Assessment  of  Crew  Workload 

One  accepted  definition  of  mental  workload  (MWL),  as  proposed  by  Young  and  Stanton  (2001), 
suggests  that  the  MWL  of  a  task  represents  the  level  of  attentional  resources  required  to  meet 
both  objective  and  subjective  performance  criteria  and  is  mediated  by  task  demand,  external 
support,  and  past  experience.  Pilot  workload  is  a  particularly  important  construct  because 
overall  mission  accomplishments  are  directly  related  to  the  resources  required  to  meet  pilot 
objectives.  Excessively  high  workload  results  in  task  shedding,  poor  decisionmaking,  and 
decreased  performance.  One  goal  of  the  human  factors  engineer  is  to  manipulate  external 
support  so  operators  are  able  to  accomplish  tasks  while  reducing  the  amount  of  resources 
required  to  accomplish  their  objectives.  As  such,  the  currently  implemented  test  design 
controlled  for  task  demand  (held  tasks  constant  across  test  conditions)  and  past  experience  (only 
used  experienced  pilots)  while  changing  external  support  (EDM  with  TAIS  symbology  vs.  no 
TAIS  symbology). 

2.5.1  Bedford  Workload  Rating  Scale  (BWRS) 

Immediately  after  each  mission,  pilots  completed  the  BWRS,  which  assesses  subjective  pilot 
workload  via  a  unidimensional  decision  tree  (appendix  A)  for  tasks  predicted  to  have  the  most 
impact  on  air  crew  workload  during  the  mission. 

The  BWRS  has  been  used  extensively  by  the  military,  civil,  and  commercial  aviation 
communities  for  pilot  workload  estimations  (Roscoe  and  Ellis,  1990).  Essentially,  it  measures 
spare  capacity  and  requires  the  participant  to  answer  a  series  of  discrete  questions  related  to 
spare  capacity  to  reach  a  specific  workload  rating.  Spare  workload  capacity  is  an  important 
resource  for  pilots  because  they  are  often  required  to  perform  several  tasks  concurrently;  for 
example,  perform  navigation  tasks,  communicate  via  multiple  radios,  aviate  the  aircraft,  and 
monitor  the  system. 

2.5.2  Visual  Workload 

Visual  workload  refers  to  the  amount  of  time  the  pilots’  visual  attention  is  diverted  from  the 
OTW  scene  to  the  instrument  cluster.  Essentially,  this  means  that  as  the  pilots  spend  more  time 
focusing  their  visual  attention  inside  the  cockpit,  the  higher  the  visual  workload.  To  quantify 
this,  a  head  and  eye  tracker  system  was  implemented  to  calculate  overall  dwell  times  for  pilots’ 
gaze  behavior.  This  data  was  collected  to  determine  how  the  addition  of  the  TAIS  symbology 
that  was  integrated  into  the  EDM  affected  how  visual  attention  was  distributed  in  the  cockpit. 
Specifically,  the  data  would  be  used  to  determine  if  TAIS  symbology  allowed  copilots  to  spend 
more  time  focusing  their  attention  OTW. 

2.6  Assessment  of  Crew  Situational  Awareness 

SA  can  be  defined  as  the  pilot’s  mental  model  of  the  current  state  of  the  flight  and  mission 
environment.  A  more  formal  definition  is  “the  perception  of  the  elements  in  the  environment 
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within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the  projection  of 
their  status  in  the  near  future”  (Endsley,  1988).  The  importance  of  SA  lies  in  its  correlation  with 
performance.  Pilots  that  report  higher  SA  also  demonstrate  better  decisionmaking  and  overall 
performance.  For  the  current  study,  increases  in  SA  should  improve  navigation  around 
unexpected  obstacles. 

2.6.1  Situation  Awareness  Rating  Technique  (SART) 

The  SART  (appendix  B)  is  a  multidimensional  rating  scale  that  captures  three  components  of 
SA:  (1)  demand  of  attentional  resources,  (2)  supply  of  attentional  resources,  and  (3) 
understanding.  Taylor  (1990)  developed  it  specifically  for  air  crew  systems  and  suggested  that 
SA  depends  on  the  pilot’s  understanding  (U)  (e.g.,  quality  of  information  they  receive)  and  the 
difference  between  the  demand  (D)  on  the  pilot’s  resources  (e.g.,  complexity  of  mission)  and  the 
pilot’s  supply  (S)  (e.g.,  ability  to  concentrate).  When  D  exceeds  S,  there  is  a  negative  effect  on 
U  and  an  overall  reduction  of  SA.  The  formula  SA  =  U  -  (D  -  S)  is  used  to  derive  the  overall 
SART  score.  According  to  Endsley  (2000),  the  SART  is  one  of  the  most  thoroughly  tested  rating 
scales  for  estimating  SA. 

2.6.2  Battlefield  Elements  SA  Questionnaire 

The  battlefield  elements  SA  questionnaire  (appendix  C)  is  a  five-point  Likert-type  scale  that 
assesses  self-reported  SA  on  a  series  of  questions;  for  example,  location  of  friendly  units, 
location  of  my  aircraft  during  the  mission,  and  location  of  other  aircraft  related  to  the  mission. 
This  questionnaire  allows  pilots  to  rate  their  own  level  of  SA. 

2.7  EDM  Interface 

The  EDM  is  a  device  currently  fielded  for  improving  SA  in  analog  cockpits.  The  purpose  of  this 
capability  is  to  enhance  SA  and  contribute  to  successful  mission  performance  by  transmitting 
near-real-time  TAIS  symbology  to  the  EDM  to  provide  important  DAUs  to  the  pilots.  To  assess 
the  effectiveness  of  the  EDM,  questionnaires  were  developed  to  allow  pilots  to  report  at  the  end 
of  each  mission  any  problems  that  contributed  to  high  workload  and  low  SA.  The  EDM 
questionnaire  addressed  the  integration  of  near-real-time  TAIS  symbology  and  is  presented  in 
appendix  D. 

2.8  Overhead  Cockpit  Cameras 

Two  over-the-shoulder  cameras  were  mounted  above  the  left  and  right  seat  to  record  pilot 
actions.  This  aided  in  determining  what  the  pilots  were  doing  during  the  different  phases  of  the 
mission.  The  cameras  had  a  time  stamp  so  that  elapsed  time  could  be  compared  with  other  data 
collected. 
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2.9  Head  and  Eye  Tracker  System 


Copilot  visual  gaze  and  dwell  times  were  collected  with  a  head  and  eye  tracking  system  from 
Applied  Science  Laboratories  (ASL).  The  ASL  eye-head  package  included  a  Model  501  eye 
tracker  and  an  Ascension  Laserbird  head  tracker.  This  system  was  primarily  selected  because  of 
its  eye-head  integration  capability,  which  allowed  unrestricted  head  movement.  This  technology 
enabled  researchers  to  collect  data  that  specified  point  of  gaze  with  respect  to  stationary  objects 
(MFDs)  within  the  cockpit.  The  ASL  software  allowed  data  collectors  to  continuously  monitor 
the  eye  position  of  the  pilots  with  crosshairs  superimposed  over  live  imagery  (figure  2).  The 
software  also  included  a  built-in  analysis  tool  that  facilitated  data  viewing  in  a  tabular  or 
graphical  format. 


Figure  2.  Eye  tracker,  pupil/camera  monitors,  and  control  panel  interface. 

2.10  Data  Analysis 

Pilot  responses  to  the  BWRS,  SART,  and  open-ended  questionnaires  were  illustrated  using  mean 
and  percentage  descriptors.  Their  responses  were  further  analyzed  with  the  Wilcoxon  signed 
rank  test  (WSRT)  to  compare  the  rating  between  the  pilot  and  copilot  and  across  each  previously 
mentioned  test  condition  to  determine  if  the  differences  were  statistically  significant  (<0.05). 

Eye  tracker  data  were  summarized  by  calculating  the  total  percentage  of  fixations  that  occurred 
for  the  different  areas  of  interest  (AOIs).  Five  AOIs  were  created  for  the  copilot:  (1)  primary 
display,  (2)  OTW,  (3)  kneeboard  and  EDM,  (4)  pilot  instrument  panel,  and  (5)  other,  the  latter 
being  eye  fixations  not  focused  on  a  specific  AOI. 

2.11  Evaluation  Limitations 

Several  limitations  must  be  considered  when  interpreting  the  data  listed  in  the  results  section. 

•  There  was  a  small  sample  size  of  pilots  ( N  =  4)  that  may  not  be  representative  of  the  total 
population. 
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•  Flight  controls  were  not  available  to  the  copilot,  as  would  be  the  case  in  a  real  life  situation. 

•  There  was  no  training  for  the  EDM. 

•  The  same  mission  scenario  was  used  for  each  vignette,  possibly  introducing  a  learning  effect. 

•  Global  positioning  system  data  did  not  consistently  synchronize  to  the  EDM  throughout  the 
event. 

These  limitations  are  not  uncommon  when  replicating  a  complex  aviation  system  in  a  simulator. 
However,  the  information  the  data  listed  in  the  results  and  summary  sections  of  this  report  should 
be  interpreted  in  the  context  of  these  limitations.  Additional  data  should  be  collected  during 
future  simulations  and  tests  to  augment  and  expand  the  findings  in  this  report. 


3.  Results 


3.1  Crew  Workload  -  Average  Bedford  Workload  (BWL)  Ratings  for  Flight  and  Mission 
Tasks 

The  overall  crew  workload  (pilot  and  copilot)  rating  for  all  aircraft  training  manual  (ATM)  tasks 
for  VI  was  3.5  (scale  ranging  1-10).  For  V2  and  V3,  the  mean  overall  crew  workload  rating  for 
all  ATM  tasks  was  2.65  and  2.67,  respectively.  All  average  BWL  scores  can  be  viewed  in 
table  3.  A  workload  rating  between  2  and  3  indicates  that  pilots  typically  had  “enough  workload 
capacity  for  all  desirable  additional  tasks,”  while  a  rating  of  3.5  indicates  that  pilots  had 
“insufficient  workload  capacity  for  easy  attention  to  additional  tasks.”  The  difference  in 
workload  between  VI  and  V2  was  statistically  significant  (WSRT,  z  =  -3.33,  p  <  0.001). 
However,  the  overall  workload  scores  between  V2  and  V3  were  not  statistically  significant 
(WSRT,  z  =  -0.05,  p  =  0.96). 

When  making  the  same  comparisons  for  pilot  and  copilot  data,  a  similar  pattern  emerged. 
Copilot  overall  workload  significantly  reduced  from  VI  to  V2  (WSRT,  z  =  -4.6,  p  <  0.001),  but 
no  significant  difference  was  detected  from  V2  to  V3  (WSRT,  z  =  -1.05,  p  >  0.05).  However, 
pilot  overall  workload  significantly  increased  from  VI  to  V2  (WSRT,  z  =  -2.24,  p  <0.05),  but 
showed  no  significant  differences  between  V2  and  V3  (WSRT,  z  =— 1.3,  p  =  0.195). 

When  investigating  average  BWL  scores  per  task  across  vignettes,  the  highest  three  scores 
should  be  addressed.  During  VI,  the  copilot  found  it  particularly  difficult  to  maintain  airspace 
surveillance  (ATM  task  1026).  An  average  score  of  6  indicates  that  “the  level  of  effort  allows 
little  attention  to  additional  tasks.”  In  addition,  the  dynamic  events  that  occurred  in  VI  caused  a 
reduction  in  spare  capacity  for  the  copilot  while  using  the  EDM.  A  score  of  5  (navigate  EDM 
pages)  indicates  that  “additional  tasks  cannot  be  given  the  desired  amount  of  attention,”  while 
reading  the  symbology  and  changing  the  map  scales  received  an  average  BWL  score  of  4, 
indicating  “insufficient  workload  capacity  for  easy  attention  to  additional  tasks.” 


8 


Table  3.  Average  BWL  scores. 


ATM  Task 

Average  Pilot 
Workload  Data 

Average  Copilot 
Workload  Data 

Overall  Workload 
Data 

VI 

V2 

V3 

VI 

V2 

V3 

VI 

V2 

V3 

1026  -  Maintain  airspace  surveillance 

2.5 

3 

2.3 

6 

3 

2.3 

4.25 

3 

2.3 

1032  -  Perform  radio  communication  procedures 

2.5 

3 

2.3 

3.5 

1.6 

2.3 

3 

2.3 

2.3 

1044  -  Navigate  by  pilotage  and  dead  reckoning 

2.5 

3 

2.6 

3.5 

2.3 

2.6 

3 

2.7 

2.6 

1054  -  Select  landing  zone/pickup  zone/holding 
area 

2.5 

3 

2.6 

3.5 

2.3 

2.6 

3 

2.7 

2.6 

2024  -  Perform  terrain  flight  navigation 

2.5 

3 

2.6 

3.5 

2.6 

2.6 

3 

2.8 

2.6 

1 146  -  Perform  electronically  aided  navigation 

— 

— 

— 

3.5 

2.3 

2.6 

— 

— 

— 

1260  -  Operate  digital  map 

— 

— 

— 

3.5 

2.6 

2.6 

— 

— 

— 

Average  across  ATM  tasks 

2.5 

3 

2.5 

3.9 

2.4 

2.5 

3.25 

2.7 

2.5 

EDM  Tasks 

Navigate  EDM 

— 

— 

— 

5 

3 

3 

— 

— 

— 

Read  EDM  symbology 

— 

— 

— 

4 

3.3 

3.6 

— 

— 

— 

Communicate  EDM  navigation  information  to  the 
pilot 

— 

— 

— 

3.5 

2 

2.6 

— 

— 

— 

Change  EDM  map  scale 

— 

— 

— 

4 

2 

3 

— 

— 

— 

3.1.1  Impact  of  Workload  on  Aircrew  Coordination 

During  VI,  the  copilot  was  required  to  spend  a  significant  amount  of  time  writing  down  grid 
coordinates  to  map  out  DAUs  that  could  not  be  preloaded  onto  the  AMPS  data  card.  Often,  the 
pilot  had  already  flown  through  the  ROZ  by  the  time  the  copilot  was  able  to  coordinate  an 
alternate  flight  path.  During  V2  and  V3,  verbal  air  crew  coordination  increased  significantly 
because  the  copilot  was  able  to  instantly  observe  the  DAU  in  reference  to  the  preplanned  flight 
and  coordinate  an  alternate  path  more  efficiently.  This  pattern  is  reflected  in  the  following 
description  of  visual  gaze  data. 

3.1.2  Visual  Workload 

Data  from  the  head  and  eye  tracker  system  were  categorized  into  the  five  AOIs  in  figure  3.  The 
OTW  AOI  includes  all  gaze  data  from  copilots  viewing  the  environmental  scene.  The 
AOI-labeled  pilot  instruments  simply  include  the  center  and  pilot  instrument  console.  The 
heads-down  data  include  both  the  EDM  and  notepad,  while  primary  display  includes  the 
instrument  panel  directly  in  front  of  the  copilot.  The  AOI-labeled  “other”  includes  all  data  that 
did  not  fall  within  one  of  the  previously  mentioned  AOIs. 
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Figure  3.  AOIs  used  to  calculate  visual  workload. 

Figure  4  shows  the  average  percentage  of  time  that  the  copilot  was  visually  focused  in  each  AOI 
during  VI.  As  can  be  seen,  the  copilot  spent  nearly  79%  of  the  time  focusing  on  the  EDM  and 
notepad  used  to  write  down  coordinates  of  dynamic  airspace  events.  Since  both  of  these  AOIs 
were  able  to  move  freely,  the  eye  tracking  system  does  not  allow  reliable  resolution  of  the  exact 
percentage  differences  between  the  notepad  and  EDM.  However,  based  on  a  review  of  the 
video,  it  was  generally  observed  that  the  copilot  spends  a  significant  and  unsafe  portion  of  time 
writing  grid  coordinates  for  dynamic  airspace  events.  Furthermore,  regardless  of  the  area  of 
fixation  (EDM  or  notepad),  both  AOIs  require  deep  heads-down  engagement. 


Figure  4.  Vignette  1  gaze  percentages. 
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Figure  5  shows  the  percentage  of  time  that  the  copilot  was  visually  focused  in  each  AOI  during 
V2.  A  fivefold  increase  in  heads-up  time  resulted  from  the  transmission  of  dynamic  airspace 
information  directly  to  the  EDM.  Although  55.85%  heads-down  time  is  still  a  significant  portion 
of  time  looking  deep  within  the  cockpit,  it  is  important  to  note  that  these  vignettes  were  designed 
to  represent  heavy  navigational  task  loading  on  the  copilot.  This  was  done  to  create  a  realistic 
environment  representative  of  the  increasingly  congested  and  dynamic  airspace. 


Figure  5.  Vignette  2  gaze  percentages. 


Figure  6  shows  the  percentage  of  time  that  the  copilot  was  visually  focused  in  each  AOI  during 
V3.  This  data  suggests  that  there  is  no  difference  in  visual  workload  between  V2  and  V3. 
However,  the  users  reported  that  annunciations  of  dynamic  events  were  helpful  indicators  that 
increased  SA.  During  this  test  event,  eye  tracker  data  were  not  sensitive  to  annunciations  of 
dynamic  events  present  in  V3. 
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Figure  6.  Vignette  3  gaze  percentages. 


3.2  Crew  Situation  Awareness 

Pilot  and  copilot  SART  scores  (table  4)  are  inconclusive  because  of  limited  statistical  power. 
Although  the  pilot’s  SART  scores  remained  relatively  constant  across  vignettes,  the  copilot’s 
SART  scores  demonstrate  a  trend  toward  improved  SA. 


Table  4.  Overall  SART  scores  across  vignettes. 


SART  Scores 

Vignette  1 

Vignette  2 

Vignette  3 

Pilot 

Participant  1 

27 

33 

16 

Participant  2 

16 

12 

16 

Participant  3 

22 

16 

28 

Average  Pilot  SART  Scores 

21.7 

20.3 

20 

Copilot 

Participant  1 

12 

23 

22 

Participant  2 

No  response 

22 

16 

Participant  3 

NA 

12 

28 

Average  Copilot  SART  Scores 

12 

19 

22 

Note:  NA  =  not  applicable. 
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Data  from  the  battlefield  elements  SA  questionnaire,  illustrated  in  table  5,  indicate  that  SA 
significantly  increased  from  VI  to  V2  but  not  from  V2  to  V3.  As  would  be  expected,  there  were 
also  significant  statistical  differences  between  VI  and  V3.  These  comparisons  were  conducted 
for  the  overall  crew  (pilot  and  copilot  data  combined)  and  per  the  pilot  and  copilot  data.  The 
same  trend  was  observed  for  all  three  comparison  sets;  for  example,  pilot  data  across  all 
vignettes,  copilot  data  across  all  vignettes,  and  overall  crew  data  across  all  vignettes.  Table  5 
illustrates  the  statistical  results  of  each  comparison. 


Table  5.  Situation  awareness  comparisons  for  crews,  pilots,  and  vignettes. 


Comparison 

Wilcoxon  Signed  Rank  Test  Results 

Crew  VI  vs.  Crew  V2 

WSRT,  z  =  -2.328,  p  =  0.02a 

Crew  V2  vs.  Crew  V3 

WSRT,  z  =  -.237,  p  >  0.05 

Crew  VI  vs.  Crew  V3 

WSRT,  z  =  -2.088,  p  =  0.037a 

Pilot  VI  vs.  Pilot  V2 

WSRT,  z  =  -2.233,  p  =  0.026a 

Pilot  V2  vs.  Pilot  V3 

WSRT,  z  =  -.25,  p  >  0.05 

Pilot  VI  vs.  Pilot  V3 

WSRT,  z  =  -2.754,  p  =  0.006a 

Copilot  VI  vs.  Copilot  V2 

WSRT,  z  =  -2.328,  p  =  0.02a 

Copilot  V2  vs.  Copilot  V3 

WSRT,  z=  -2.088,/;  =  0.813 

Copilot  VI  vs.  Copilot  V3 

WSRT,  z  =  -3.334,  p  =  0.001a 

dStatistically  significant  comparisons. 


3.3  Human  Factors  Observations 

During  each  vignette,  a  human  factors  SME  observed  pilots  from  the  battle  captain  station 
(figure  7),  where  observers  can  monitor  a  video  feed  of  the  aircraft  in  its  synthetic  environment, 
a  video  feed  of  pilot  EDM  activity,  an  over-the-shoulder  video  feed  of  the  pilots,  and  voice 
communications  between  pilots  and  the  battle  captain.  Throughout  the  vignettes,  several  key 
observations  were  made  in  terms  of  performance,  air  crew  communication,  and  EDM  interface 
activity.  These  observations  are  noted  in  section  3.3.1. 


Figure  7.  Battle  captain  station. 
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3.3.1  Flight  Path  Performance 

In  VI,  the  copilot  spent  a  significant  portion  of  heads-down  time  attempting  to  map  out  dynamic 
ACM  events.  Often  they  were  unable  to  gain  SA  on  these  events  quickly  enough  to 
communicate  an  alternate  route  to  the  pilot.  Consequently,  the  air  crew  frequently  flew  directly 
through  an  ROZ  without  being  aware  of  where  exactly  the  danger  was  located  until  after  they 
had  passed  it.  This  significant  performance  risk  was  mitigated  by  transmitting  DAUs  to  the 
EDM  in  V2  and  V3. 

3.3.2  Air  Crew  Navigation  Coordination 

As  previously  mentioned,  the  copilots  spent  much  of  their  time  writing  grid  coordinates  during 
VI.  Compared  to  V2  and  V3,  there  was  more  specific  navigational  instruction  from  copilot  to 
pilot.  For  example,  the  copilot  would  instruct  the  pilot  to  bank  left  10°  to  move  around  the 
airspace  of  a  UAV.  This  type  of  air  crew  coordination  occurred  more  frequently,  introducing 
dynamic  ACMs  to  the  EDM  and,  thus,  enabling  the  copilot  to  coordinate  navigation  information 
to  the  pilot  more  efficiently. 

3.3.3  EDM  Interface  and  Usage 

Introducing  additional  symbology  into  the  EDM  display  resulted  in  more  occlusion  and  symbol 
clutter.  In  several  instances,  symbology,  either  from  TAIS  or  AMPS,  occluded  text  or  was  too 
saturated  against  a  similarly  colored  background  for  the  user  to  read.  During  operations,  the  user 
sometimes  had  to  rotate  the  EDM  to  read  text  or  hold  the  device  up  for  the  pilot  to  briefly  view. 
For  the  current  study,  the  EDM  was  not  mounted  to  the  cockpit,  which  influenced  its  usage  and 
sharing  of  information.  However,  pilots  often  mount  the  EDM,  which  restricts  their  ability  to 
rotate  and  share  information  displayed  on  the  device.  Typically,  both  pilots  would  have  an 
EDM,  but  because  of  the  variety  of  overlays  and  map  scales,  it  is  often  easier  for  the  copilot  to 
show  the  pilot  the  EDM  interface  to  gain  shared  SA  rather  than  referencing  information  that  is 
displayed  differently  on  each  device. 

During  V2,  symbology  from  TAIS  was  injected  into  the  EDM  without  an  annunciation.  In  this 
case,  symbology  was  often  located  in  a  position  outside  of  the  current  viewing  scale  of  the  map. 
Thus,  pilots  were  unaware  of  the  new  information  until  it  came  within  closer  range  of  the  aircraft 
or  the  operator  of  the  EDM  happened  to  scale  out  of  the  map  far  enough  to  see  the  newly 
introduced  ACM.  In  V3,  this  issue  was  mitigated  by  the  annunciation,  which  enabled  pilots  to 
plan  farther  in  advance. 

In  addition,  when  writing  on  the  digital  notepad,  the  copilot  often  had  to  restart  writing 
characters  because  the  interface  did  not  consistently  detect  the  user  input. 
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3.4  Crew  Open-Ended  Comments 

Throughout  each  questionnaire,  pilots  were  asked  to  provide  open-ended  feedback  to  various 
questions  regarding  workload,  SA,  and  the  EDM.  The  most  consistently  expressed  issues  are 
discussed  in  the  following  paragraphs.  The  raw  data  is  presented  in  appendices  E  and  F. 

3.4.1  Workload  Open-Ended  Comments 

Several  primary  themes  emerged  from  the  open-ended  workload  comments.  First,  there  was  a 
lack  of  EDM  training.  In  addition,  workload  was  significantly  increased  when  the  EDM  data 
was  not  syncronizing  properly.  These  data  suggest  that  workload  will  increase  to  unmanageable 
levels  in  the  event  of  EDM  failure. 

3.4.2  SA  Open-Ended  Comments 

Open-ended  SA  comments  revealed  three  issues.  First,  during  technical  difficulties  in  which 
data  did  not  properly  sync  with  the  EDM,  pilots  consistently  reported  low  SA.  These  responses 
suggest  pilot  reliance  on  EDM  technology  and  its  overall  effectiveness  in  providing  SA.  Second, 
several  pilots  pointed  out  the  difficulty  in  predicting  UAV  headings.  This  information  is 
particularly  important  for  alternate  course  of  action  route  planning.  Third,  as  mentioned 
previously,  text  was  difficult  for  pilots  to  read  and  ROZs  were  not  transparent  enough  to  see  any 
other  information  within  the  ROZ  symbology. 

3.4.3  EDM  Open-Ended  Comments 

The  overall  EDM  responses  point  to  the  benefits  and  issues  outlined  in  table  6. 


Table  6.  EDM  responses. 


Unfavorable  Responses 

Favorable  Responses 

The  notepad  was  hard  to  read  and  write  on. 

The  digital  map  and  route  overlays  were  beneficial. 

The  device  was  too  hot  to  keep  strapped  on  their  leg  for 
extended  periods. 

The  dynamic  airspace  changes  provided  excellent  SA. 

ACMs  not  transparent  enough. 

The  moving  map  capability  was  helpful. 

Lack  of  customization. 

The  “next  ACP  information"  function  helped  to  inform 

navigation  decisions. 

Inability  to  pan  maps. 

The  “look  forward,”  “look  backward,"  and  “direct  too” 

map  navigation  functions  worked  well. 

Font  size  was  not  big  enough. 

Enabled  us  to  maintain  airspace  surveillance. 

Text  was  sometimes  upside  down. 

ROZs  provided  excellent  navigation  information. 

Unable  to  predict  UAV  heading. 

— 

Symbols  often  blended  into  the  map. 

— 

Maps  were  slow  to  load. 

— 

Lack  of  EDM  training. 

— 
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Overall,  these  responses  support  the  following  notions.  First,  the  EDM  is  an  excellent  source  of 
SA  for  the  pilots,  particularly  when  dynamic  ACMs  are  automatically  pushed  to  the  device. 
Second,  there  are  several  graphical  user  interface  issues  on  the  EDM  that  need  to  be  addressed, 
particularly  with  the  addition  of  more  symbology  to  the  interface.  Font  size,  text  orientation,  and 
symbol  clutter  (ACM  transparency)  are  all  issues  that  will  require  further  development  and 
testing  to  properly  integrate  the  EDM  into  the  cockpit. 


4.  Conclusions 


4.1  Crew  Workload 

The  overall  trend  of  the  BWRS  data  indicates  a  continuous  reduction  of  workload  from  VI  to  V3 
for  both  pilot  and  copilot.  However,  the  magnitude  of  the  reduction  was  the  largest  for  the 
copilot  and  from  VI  to  V2.  There  were  further  reductions  in  workload  from  V2  to  V3  but  they 
were  not  as  pronounced  as  the  first  comparison.  A  similar  trend  was  reflected  when  comparing 
the  effects  of  DAUs  on  the  reduction  of  pilot  workload.  While  the  copilot  experienced  the 
greater  effect,  the  pilot  was  affected,  demonstrating  that  the  information  that  was  initially 
presented  only  to  the  copilot  had  a  measurable  effect  on  the  other  crew  member. 

Pilots  reported  that  they  typically  experienced  tolerable  workloads  when  performing  all  missions. 
However,  all  peak  workload  scores  were  reported  during  V 1 ,  which  represented  the  current 
capability,  in  which  the  copilot  was  required  to  mentally  transform  lattitude/longitude  data  to 
gain  SA  about  DAUs.  By  comparison,  the  workload  scores  obtained  during  V2  and  V3  indicate 
a  substantial  reduction  in  crew  workload  attributable  to  integrating  dynamic  airspace  symbology 
from  TAIS  into  the  cockpit. 

The  highest  average  BWRS  score  was  a  6  on  ATM  task  1026  (maintain  airspace  surveillance), 
and  was  produced  by  copilots  indicating  that  “the  level  of  effort  allowed  for  little  attention  to 
additional  tasks.”  The  consequence  of  this  unmanageable  level  of  workload  is  that  the  copilot 
cannot  adequately  inform  the  pilot  of  flight  route  alterations.  On  several  occasions,  the  pilot 
flew  directly  through  several  ROZs  without  any  warning  of  the  potential  danger.  This  is  a 
serious  risk  that  was  mitigated  by  the  support  of  dynamic  information  being  automatically 
transmitted  into  the  cockpit. 

The  remaining  three  of  the  highest  BWRS  scores  were  all  in  reference  to  EDM  usage,  which  can 
be  attributed  to  pilot  familiarity  with  the  EDM  and  symbol  clutter  issues  (including  font  size  and 
text  orientation).  Navigating  the  EDM  and  changing  the  EDM  map  scales  initially  received 
higher  than  average  scores  but  deflated  across  vignettes.  This  is  most  likely  due  to  increased 
familiarity  with  the  device  because  the  introduction  of  dynamic  TAIS  symbology  in  V2  and  V3 
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had  no  effect  on  the  mechanics  of  operating  the  EDM.  However,  BWRS  scores  for  reading 
symbology  on  the  EDM  remained  higher  than  average  across  all  vignettes.  This  is  due  to  the 
increase  in  symbol  clutter  from  the  addition  of  TAIS  symbology.  The  solution  to  this  problem  is 
complex  and  will  require  further  study  to  optimize  the  presentation  of  numerous  and  complex 
symbology  on  a  dynamic  background. 

4.2  Visual  Workload 

Visual  workload,  i.e.,  eye  and  head  tracker  data,  indicated  unacceptable  visual  demands  on  the 
copilot  during  VI.  Throughout  VI,  78.73%  of  the  copilots’  time  was  spent  gazing  at  the  notepad 
and  EDM,  while  only  5.34%  of  their  time  was  spent  maintaining  airspace  surveillance.  This  left 
the  task  of  airspace  surveillance  almost  solely  up  to  the  pilot,  which  means  that  every  time  the 
pilot  looked  down  (typically  15%  to  39%  of  the  total  flight  time,  according  to  Havir  et  al.,  2006) 
there  is  a  high  probability  that  neither  crew  member  was  looking  outside  the  window. 

During  V2  and  V3,  visual  workload  was  reduced  significantly  compared  to  VI.  The  OTW 
viewing  times  increased  substantially  from  5.34%  in  VI  to  24.29%  in  V2  and  to  27.68%  in  V3. 
This  indicates  that  there  was  a  considerable  “release”  of  visual  capture  with  the  introduction  of 
dynamic  ACM  information.  Results  obtained  in  V2  and  V3  were  consistent  with  copilot  OTW 
viewing  times  reported  during  other  simulation  tests  (Havir  et  al.,  2006).  Although  a  statistical 
analysis  was  not  conducted  on  this  data  because  of  a  low  sample  size,  visual  workload  appears  to 
have  remained  constant  across  V2  and  V3,  indicating  that  the  increase  in  SA  and  slight  reduction 
in  BWRS  scores  from  V2  to  V3  occurred  with  no  additional  cost  of  visual  workload. 

4.3  Crew  Situational  Awareness 

Overall,  the  SART  and  battlefield  elements  SA  data  indicate  that  SA  was  significantly  increased 
by  the  integration  of  dynamic  ACM  events  into  the  cockpit.  These  results  can  be  interpreted 
based  on  Endsley’s  (1995b)  three  stages  of  SA  (perception,  comprehension,  and  projection). 

4.3.1  Level  1  Situational  Awareness  (Perception) 

During  VI,  copilots  failed  to  consistently  gain  timely  perception  of  dynamic  ACM  events. 

Events  in  which  flight  routes  were  successfully  altered  based  on  new  information  were  too  taxing 
for  the  copilot  to  maintain  airspace  surveillance.  Thus,  the  cost  incurred  by  the  copilot,  an 
inconsistent  gain  in  basic  perceptual  information  about  important  airspace  events,  was 
unacceptably  high.  This  decreases  performance  and  indicates  that  critical  incidences  (incidences 
of  fratricide  and  controlled  flight  into  terrain  [CFIT])  will  become  more  probable  as  the  airspace 
becomes  more  congested.  Comprehension  and  prediction  of  airspace  elements  was  minimal  in 
VI. 

4.3.2  Level  2  and  3  Situational  Awareness  (Comprehension  and  Prediction) 

During  V2,  copilots  were  capable  of  gaining  perceptual  SA  in  an  efficient  manner. 

Consequently,  because  of  the  nature  in  which  the  information  was  presented  (spatial  information 
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presented  visually  as  opposed  to  verbally),  they  were  able  to  comprehend  multiple  sources  of 
information  and  gain  an  overall  mission  picture  that  allowed  pilots  to  predict  and  then  alter  their 
flight  path.  Moreover,  crew  access  to  dynamic  airspace  information  related  to  environmental 
changes  and  the  commander’s  response  to  those  changes  (reflective  of  intent)  provide  the 
building  blocks  for  situational  understanding.  Without  a  near-real-time  airspace  picture,  pilots 
will  not  be  able  to  gain  situational  understanding  while  operating  in  a  dynamic  environment. 
Essentially,  the  same  effect  occurred  during  V3  but  the  addition  of  an  annunciation  provided 
more  consistent  and  timely  perception  of  airspace  elements.  The  increased  consistency  of 
perceiving  the  newly  introduced  elements  amplified  comprehension  of  dynamic  airspace  events. 
That  consistency,  combined  with  an  increase  in  timeliness,  supported  the  predictability  of 
airspace  events. 

4.4  Electronic  Data  Management  Interface 

As  an  interface  for  displaying  airspace  symbology,  the  EDM  was  widely  accepted  by  the  user 
group,  particularly  for  its  basic  map  overlays,  route  display,  and  general  navigation  support. 
However,  several  issues  must  be  resolved  that  relate  to  alterations  to  the  device  with  the  addition 
of  DAUs,  for  example,  how  dynamic  airspace  symbology  is  presented  (font  size,  orientation,  and 
clutter  issues)  and,  more  generally,  EDM  usage  issues  unrelated  to  the  introduction  of  dynamic 
airspace  symbology.  Pilots  also  reported  that  the  device  became  too  hot  to  strap  onto  their  legs, 
a  pan  feature  should  be  added,  the  notepad  was  inadequate,  and  the  device  was  too  slow  when 
updating  maps  and  symbology.  Further  software  development  should  mitigate  the  interface 
issues,  while  mounting  the  EDM  to  the  cockpit  would  alleviate  the  heat  issue. 

4.4.1  Dynamic  Airspace  Integration  Issues:  Symbol  Clutter/Saturation  and  Transparency 

The  primary  concerns  of  introducing  dynamic  airspace  symbology  into  the  EDM  are  symbol 
clutter  and  transparency.  Currently,  if  symbols  overlap  each  other,  at  least  one  will  be  occluded; 
often  both  become  illegible.  To  mitigate  this  issue,  the  EDM  user  should  be  able  to  select  a 
cluster  of  symbols  and  distribute  them  so  that  the  user  can  view  each  symbol  related  to  the  same 
overlapping  area  and  select  it  to  gain  further  information,  such  as  altitudes  and  expiration.  ROZ 
transparency  also  needs  to  be  altered  so  that  text  or  symbology  within  the  ROZ  is  still  detectable. 
In  addition,  in  some  instances,  symbols  were  masked  because  of  the  similarity  of  the  symbol  hue 
and  map  overlay  hue.  A  hue-correlation  detection  algorithm  could  be  introduced  to  correlate 
symbol  hue  with  the  currently  selected  map  overlay  and  alter  the  symbol  hue  to  a  color  that  is 
easily  viewed  in  that  particular  map  overlay.  This  is  a  particularly  complex  problem  requiring 
further  development  and  testing. 

4.5  Summary  of  Conclusions 

Rotary- wing  assets  with  analog  cockpits  currently  rely  on  radio  communications  for  dynamic 
airspace  information.  As  such,  the  current  method  of  transmitting  information  into  the  cockpit 
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requires  an  increase  in  overall  crew  workload  (particularly  for  the  copilot),  creates  extensive 
visual  demands  that  divert  the  copilot’s  visual  attention  away  from  OTW,  and  hinders  SA  and 
the  development  of  situational  understanding.  The  consequences  of  these  measures  indicate  that 
in  the  increasingly  congested  airspace,  a  higher  probability  of  fratricide  and  CFIT  incidence  will 
likely  occur  unless  a  mitigation  strategy  is  implemented  to  augment  the  capabilities  of  the  system 
to  match  the  requirements  of  the  current  and  future  airspace  environment.  To  meet  these  needs, 
the  current  test  simulated  the  integration  of  near-real-time  TAIS  symbology  into  the  cockpit  and 
measured  its  effects.  Overall,  copilots  experienced  a  reduction  in  workload  because  of  the 
efficiency  in  which  dynamic  information  was  presented,  a  decrease  in  visual  workload  because 
visual  capture  was  released  for  OTW  viewing,  and  increased  SA  because  of  the  mode  in  which 
information  was  more  efficiently  presented,  i.e.,  spatial  information  in  a  spatial  display  as 
opposed  to  spatial  information  transmitted  verbally. 


5.  Point  of  Contact 


Questions  regarding  this  report  should  be  directed  to:  Mr.  M.  Sage  lessee,  ARL-HRED, 
AMCOM  Field  Element,  at  DSN  788-8830  (michael.sage.jessee@us.army.mil),  or  Dr.  Anthony 
Morris  at  DSN  788-9556  (tony.w.morris@us.army.mil). 
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Appendix  A.  Bedford  Workload  Decision  Tree 


This  appendix  appears  in  its  original  form,  without  editorial  change. 


Workload  insignificant 

1 

Workload  low 

2 

Enough  workload  capacity  for  all 
desirable  additional  tasks 

3 

Is  workload 
satisfactory 
without  reduction  in 
spare  (workload)  capacity'? 


I  N0  I 


I 

I 


Is  it  possible  to 
complete  the  task  ? 


Insufficient  workload  capacity  for  easy 
attention  to  additional  tasks 


Reduced  workload  capacity.  Additional 
tasks  cannot  be  given  the  desired  amount 
of  attention 


Little  workload  capacity  level  of  effort 
allows  little  attention  to  additional  tasks 


Very  little  spare  capacity,  but 
maintenance  of  effort  in  the  primary 
tasks  not  in  question 


Very  high  workload  with  almost  no 
spare  capacity  Difficulty  in  maintaining 
level  of  effort 


Extremely  high  workload  No  spare 
capacity  Serious  doubts  as  to  ability  to 
maintain  level  of  effort 


Task  abandoned.  Pilot  unable  to  apply 
sufficient  effort 


Pilot  Decisions 
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Appendix  B.  Situational  Awareness  Rating  Technique 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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SA1.  Situation  Awareness  is  defined  as  “timely  knowledge  of  what  is  happening  as  you 
perform  your  tasks  during  the  mission.” 


Situation  Awareness  Rating  Technique  (SART) 

DEMAND 

Instability  of  Situation 

Likeliness  of  situation  to  change  suddenly. 

Variability  of  Situation 

Number  of  variables  which  require  your  attention 

Complexity  of  Situation 

Degree  of  complication  (number  of  closely  connected  parts)  of  the  situation 

|  SUPPLY  ! 

Arousal 

Degree  to  which  you  are  ready  for  activity;  ability  to  anticipate  and  keep  up 
with  the  flow  of  events 

Spare  Mental  Capacity 

Amount  of  mental  ability  available  to  apply  to  new  tasks 

Concentration 

Degree  to  which  your  thoughts  are  brought  to  bear  on  the  situation;  degree  to 
which  you  focused  on  important  elements  and  events 

Division  of  Attention 

Ability  to  divide  your  attention  among  several  key  issues  during  the  mission; 
ability  to  concern  yourself  with  many  aspects  of  current  and  future  events 
simultaneously 

|  UNDERSTANDING  ! 

Information  Quantity 

Amount  of  knowledge  received  and  understood 

Information  Quality 

Degree  of  goodness  or  value  of  knowledge  communicated 

Familiarity 

Degree  of  acquaintance  with  the  situation 

Rate  the  level  of  each  component  of  situation  awareness  that  you  had.  Circle  the 
appropriate  number  for  each  component  of  situation  awareness  (e.g.,  complexity  of 
situation). 

DEMAND: 

Instability  of  situation:  (low  =  stability)  Low  1 . 2 . 3 . 4 . 5 . 6 . 7  High 

Variability  of  situation:  (low  =  few  variables)  Low  1 . 2 . 3 . 4 . 5 . 6 . 7  High 

Complexity  of  situation:  (low  =  less  complexity)  Low  1 . 2 . 3 . 4 . 5 . 6 . 7  High 

SUPPLY: 

Arousal:  (low  =  not  aroused) 

Spare  mental  capacity:  (low  =  small  amount) 

Concentration:  (low  =  unfocused  on  situation) 

Division  of  attention:  (low  =  no  multitasking) 

UNDERSTANDING: 

Information  quantity:  (low  =  little  info) 

Information  quality:  (low  =  less  value) 

Lamiliarity:  (low  =  novel) 


Low  1 . 2 . 3 . 4 . 5 . 6 . 7  High 

Low  1 . 2 . 3 . 4 . 5 . 6 . 7  High 

Low  1 . 2 . 3 . 4 . 5 . 6 . 7  High 

Low  1 . 2 . 3 . 4 . 5 . 6 . 7  High 


Low  1 . 2 . 3 . 4 . 5 . 6 . 7  High 

Low  1 . 2 . 3 . 4 . 5 . 6 . 7  High 

Low  1 . 2 . 3 . 4 . 5 . 6 . 7  High 
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Appendix  C.  Copilot  Open-Ended  Survey  Responses 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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SA2.  Rate  the  level  of  situational  awareness  you  had  for  each  of  the  battlefield  elements 
during  the  mission  by  placing  and  X  in  the  appropriate  column  for  each  battlefield 
element. 


Battlefield 

Elements 

Very  High 
Level  of 
Situation 
Awareness 

Fairly  High 
Level  of 
Situation 
Awareness 

Intermediate 
Level  of 
Situation 
Awareness 

Fairly  Low 
Level  of 
Situation 
Awareness 

Very  Low 
Level  of 
Situation 
Awareness 

Location  of 
Friendly  Units 

Location  of 

My  Aircraft 
During 

Mission 

Location  of 
Other 

Aircraft 

Related  to  the 
Mission 

Location  of 
Cultural 
Features 
e.g..  Bridges 

Route 

Information 
(ACPs,  BPs, 
EAs,  RPs) 

Describe  any  instances  when  you  feel  you  had  low  situational  awareness  during  the 
mission: 
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Appendix  D.  Electronic  Data  Manager  (EDM)  Questionnaire 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Please  answer  the  below  questions  honestly  based  on  your  experience  conducting  these 
mission  excursions. 

1.  Did  you  have  adequate  information  to  aid  the  pilot  in  navigation  tasks? 

If  yes,  please  explain 


If  no,  please  explain 


2.  How  did  the  EDM  support  your  need  for  navigational  information? 


3.  Explain  how  the  EDM  visual  information  clarified  verbal  navigation  between  copilot  and 
pilot? 


4.  Explain  how  you  used  the  change  of  scale  EDM  map  feature? 


5.  Explain  how  radio  transmitted  information  helped  you  in  your  mission  (Irrelevant  for 
Excursion  2) 
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6.  How  quickly  did  you  observe  EDM  dynamic  visual  notifications  (Irrelevant  for 
Excursion  1)? 


7.  Which  ACM  icons  communicated  the  MOST  readily  available  navigation  information? 

8.  Which  ACM  icons  communicated  the  LEAST  readily  available  navigation  information? 


9.  Please  provide  any  other  comments  regarding  how  the  EDM  supported  your  mission? 


10.  Please  name  3  likes  and  3  dislikes  regarding  your  experience  with  the  EDM? 
3  Ups 


3  Downs 
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Intentionally  lelt  blank. 
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Appendix  E.  Copilot  Open-Ended  Survey  Responses 


This  appendix  appears  in  its  original  form,  without  editorial  change. 


VIGNETTE  1 

Workload  Comments 

1.  If  you  gave  a  workload  rating  of  ‘6’  or  higher  for  any  task,  explain  why  the  workload  was  high 
for  that  task: 

•  EDM  was  acting  up,  GPS  data  kept  going  out 

•  I  was  unable  to  plot  SA  on  the  map.  There  was  a  lack  of  EDM  training.  Also,  TOH  given 
in  Lat/Long,  Grid  in  MGRS  not  able  to  compare  grid  quick  enough 

2.  In  the  mission  you  just  flew,  list  any  flight  and/or  mission  tasks  that  you  had  to  ask  your 
crewmember  to  accomplish  because  you  workload  was  too  high: 

•  Gave  him  the  route  card  to  focus  on  restarting  the  EDM  or  sinking  the  data 

Situational  Awareness  Comments 

1.  Describe  any  instances  when  you  feel  you  had  low  situational  awareness  during  the  mission: 

•  There  were  no  ROZs  displayed  and  it  is  hard  to  interpret  the  grid  on  a  moving  map 

•  The  radio  uploader  had  limited  information  associated  with  the  effective  ROZ  and  I  was 
unable  to  plot  it  on  the  EDM 

EDM  Related  Comments 

1 .  Did  you  have  adequate  information  to  aid  the  pilot  in  navigation  tasks? 

Yes,  please  explain. . . 

•  Route  overlays  are  great 

•  Route  information  was  detailed  enough  to  stay  on  schedule  to  get  to  LZs 
No,  please  explain. . . 

•  No  Responses 

2.  How  did  the  EDM  support  your  need  for  navigational  information? 

•  Training  target  graphics  (TTG)  needs  to  be  more  accurate  and  stable 

•  Route  color  and  symbols  were  good.  EDM  was  too  slow  to  upload 

3.  Explain  how  the  EDM  visual  information  clarified  verbal  navigation  between  copilot  and 
pilot? 

•  It  is  better  when  you  know  where  to  go  and  where  we  are.  It  then  becomes  easier  to 
correct  the  navigation 

•  Provided  guidance  on  the  course  to  the  LZ  and  along  the  route 

4.  Explain  how  you  used  the  change  of  scale  EDM  map  feature? 

•  To  maintain  SA  and  look  ahead,  or  look  for  more  detail 

•  I  didn’t  know  how  to  use  the  change  of  scale  EDM  map  feature.  It  really  needs  a  tactical 
map. 

5.  Explain  how  radio  transmitted  information  helped  you  in  your  mission? 

•  Gave  me  some  info  but  it  added  to  workload  by  requiring  transformation  to  moving  map 

•  It  told  me  of  hazards,  but  not  to  the  detail  that  was  needed 
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6.  Which  ACM  icons  communicated  the  MOST  readily  available  navigation  information? 

•  Some  of  the  preloads  were  good 

•  Control  Zone 

•  Couldn’t  tell  if  RZ  3 A  was  blue? 

7.  Which  ACM  icons  communicated  the  LEAST  readily  available  navigation  information? 

•  BFT  icons 

8.  Please  provide  any  other  comments  regarding  how  the  EDM  supported  your  mission? 

•  Good  to  have  a  pigmap 

•  Navigation 

3  Ups: 

•  Good  maps 

•  Graphic  capability 

•  Messages 

•  Route 

•  ACPs 

•  Maps 
3  Downs: 

•  No  top  level  data 

•  Lack  of  customization 

•  Too  much  heat 

•  Control  Zones  were  not  transparent  enough 

•  Updates  were  too  slow 

•  Not  enough  meta  data 

1 1.  In  what  ways  did  the  digital  note  pad  on  the  EDM  assist  or  hinder  you? 

•  It  is  a  little  hard  to  read  and  write  on 

•  Didn’t  use  it 

12.  Compared  to  a  paper  and  pencil  pad,  describe  the  benefits  and  shortcomings  of  the  EDM  note 
pad. 

•  Didn’t  use  it 

13.  What  additional  map  navigation  features  beyond  “look  ahead”  “look  back”  and  “zoom” 
would  you  add  to  the  EDM? 

•  Slew  -  drag  and  drop 

•  Add  an  SA  look  up  feature  to  publish  a  grid  to  see  its  location  to  the  map/route 

14.  How  did  the  map  navigation  features  support  your  decisions  and  actions  regarding  the 
dynamic  ACM  symbols? 

•  It  helped  to  keep  us  out  of  the  ACM 
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VIGNETTE  2  COMMENTS 

Workload  Comments 

1 .  If  you  gave  a  workload  rating  of  ‘6’  or  higher  for  any  task,  explain  why  the  workload  was  high 
for  that  task: 

•  No  responses 

2.  In  the  mission  you  just  flew,  list  any  flight  and/or  mission  tasks  that  you  had  to  ask  your 
crewmember  to  accomplish  because  you  workload  was  too  high: 

•  No  responses 

Situational  Awareness  Comments 

1.  Describe  any  instances  when  you  feel  you  had  low  situational  awareness  during  the  mission: 

•  During  the  mission  the  EDM  lost  position  info  causing  the  crew  to  revert  to  pilotege/dead 
reckoning  for  approx  1.5  minutes 

•  It  was  difficult  to  anticipate  the  direction  of  UAV  travel 

•  Low  SA  when  the  EDM  was  updating  slowly 

•  More  training  on  symbols  and  EDM  would  have  generally  increased  SA 

EDM  Related  Comments 

1.  Did  you  have  adequate  information  to  aid  the  pilot  in  navigation  tasks? 

Yes,  please  explain. . . 

•  Yes,  all  navigation  data  provided  allowed  for  successful  mission  accomplishment 

•  Route  information  and  ACMs  provided  enough  info  to  perform  mission 

•  All  information  was  adequate 
No,  please  explain. . . 

•  No  Responses 

2.  How  did  the  EDM  support  your  need  for  navigational  information? 

•  Provided  necessary  data  to  perform  electronically  aided  navigation 

•  Provided  route  data 

•  Provided  SA 

3.  Explain  how  the  EDM  visual  information  clarified  verbal  navigation  between  copilot  and 
pilot? 

•  Able  to  share  visual  data  to  formulate  alternate  COAs  if  required 

•  Able  to  show  ACMs  to  verify  any  decision  to  alter  planned  rout 

4.  Explain  how  you  used  the  change  of  scale  EDM  map  feature? 

•  Carefully,  it  didn’t  want  to  cooperate 

5.  How  quickly  did  you  observe  EDM  dynamic  visual  notifications? 

•  Almost  immediately 

•  I  believe,  fairly  quickly  after  they  were  displayed 

•  Possibly  need  an  audio  alert 
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6.  Which  ACM  icons  communicated  the  MOST  readily  available  navigation  information? 

•  ROZ,  corridors,  ect. . . 

•  ROZ,  ACM  used  to  alter  routes  as  necessary 

•  Control  Zones 

7.  Which  ACM  icons  communicated  the  LEAST  readily  available  navigation  information? 

•  UAV  symbols  too  slow  on  update 

•  Small  UAV  icons  moved  too  slow/updated  too  slow  to  reposition  aircraft  if  necessary 

•  I  couldn’t  tell  the  direction  of  travel  of  the  UAVs 

8.  Please  provide  any  other  comments  regarding  how  the  EDM  supported  your  mission? 

•  During  change  of  mission,  I  was  able  to  move  forward  to  verify  ACP  location  to  navigate 
direct  to 

•  Good  for  navigation 

9.  Please  name  3  likes  and  3  dislikes  regarding  your  experience  with  the  EDM? 

3  Ups: 

•  ACP  selection  for  direct  to  navigation 

•  Dynamic  ACMs  displayed 

•  Route  info 

•  Overall  mission  picture 

•  Flight  route 

•  Color  of  ROZ  was  good 
3  Downs: 

•  No  pan  feature 

•  Lack  of  other  map  scales  (simulation  issue) 

•  Too  much  heat  generated  on  the  leg 

•  Too  much  heat 

•  Inability  to  pan  map 

•  Font  size  on  some  of  the  text  is  too  small  (this  was  inconsistent  throughout  the  mission) 

•  Text  was  upside  down 

•  UAV  didn’t  indicate  direction  of  flight 

•  Symbols  blended  into  map 

10.  In  what  ways  did  the  digital  note  pad  on  the  EDM  assist  or  hinder  you? 

•  Unreadable 

•  Not  legible  and  didn’t  work  properly 

11.  Compared  to  a  paper  and  pencil  pad,  describe  the  benefits  and  shortcomings  of  the  EDM  note 
pad. 

•  Did  not  use 

•  Did  not  use 

•  No  benefit  and  not  legible 
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12.  What  additional  map  navigation  features  beyond  “look  ahead”  “look  back”  and  “zoom” 
would  you  add  to  the  EDM? 

•  Pan/Direct  to  feature 

•  Next  ACP  information 

•  Pan  (grab  and  drag  map,  just  like  google  earth) 

13.  How  did  the  map  navigation  features  support  your  decisions  and  actions  regarding  the 
dynamic  ACM  symbols? 

•  Allowed  me  to  adjust  routes  as  required  to  complete  the  mission 

•  Provided  good  SA  for  rerouting  missions 

•  Allowed  me  to  look  ahead  to  see  if  any  ACMs  would  interfere  with  the  route  of  the  flight 

VIGNETTE  3  COMMENTS 

Workload  Comments 

1 .  If  you  gave  a  workload  rating  of  ‘6’  or  higher  for  any  task,  explain  why  the  workload  was  high 
for  that  task: 

•  No  Responses 

2.  In  the  mission  you  just  flew,  list  any  flight  and/or  mission  tasks  that  you  had  to  ask  your 
crewmember  to  accomplish  because  you  workload  was  too  high: 

•  GPS  lost  data,  pilot  had  to  take  on  airspace  overview  until  EDM  was  synced 

Situational  Awareness  Comments 

1.  Describe  any  instances  when  you  feel  you  had  low  situational  awareness  during  the  mission: 

•  Friendly  forces  jumped  around  a  bit  on  the  EDM 

•  Transparency  and  some  lettering  were  hard  to  see  or  see  through.  Need  meta  drop  down 
boxes 

EDM  Related  Comments 

1.  Did  you  have  adequate  information  to  aid  the  pilot  in  navigation  tasks? 

Yes,  please  explain. . . 

•  The  moving  map  was  good 

•  Dynamic  changes  provided  the  needed  SA 

•  Yes,  route  info  and  appearance  of  ACMs  made  adjustments  to  routes  easy 
No,  please  explain. . . 

•  I  could  use  more  top  level  data.  For  example  TTG  and  digital  target  graphics  (DTG) 

2.  How  did  the  EDM  support  your  need  for  navigational  information? 

•  Could  be  even  better 

•  GPS  was  lost  for  5  minutes,  which  reduced  SA 

•  Yes,  it  provided  current  position  data  and  route  info 
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3.  Explain  how  the  EDM  visual  information  clarified  verbal  navigation  between  copilot  and 
pilot? 

•  It  is  good,  but  there  could  be  more 

•  EDM  was  easy  to  read  from  both  seats,  with  the  exception  of  text.  Copilot  had  to  read 
text  to  pilot 

•  Used  EDM  to  show  pilot  on  controls  the  situation  periodically  throughout  the  flight 

4.  Explain  how  you  used  the  change  of  scale  EDM  map  feature? 

•  In  order  to  look  ahead 

•  Supported  different  mission  sets 

•  Did  not  use 

5.  Explain  how  radio  transmitted  information  helped  you  in  your  mission. 

•  Too  much  talking 

•  It  helped  with  SA  for  ACMs 

•  Confirmed  that  ACM  had  appeared  on  EDM 

6.  How  quickly  did  you  observe  EDM  dynamic  visual  notifications? 

•  Very  quickly 

•  Quickly  enough 

•  Almost  immediately 

7.  Which  ACM  icons  communicated  the  MOST  readily  available  navigation  information? 

•  ROZ  and  ACM  for  RTE 

•  Routes  and  ROZs 

•  ROZs 

8.  Which  ACM  icons  communicated  the  LEAST  readily  available  navigation  information? 

•  BFT,  due  to  lag 

•  AH-64s  and  UAVs 

•  UAV  symbols 

9.  Please  provide  any  other  comments  regarding  how  the  EDM  supported  your  mission? 

•  It  provided  me  what  I  need  to  complete  the  mission,  but  still  could  be  streamlined  a  bit 
more 

•  Position  data  of  the  UAVs  were  good 

10.  Please  name  3  likes  and  3  dislikes  regarding  your  experience  with  the  EDM? 

3  Ups: 

•  Routes  and  ACPS 

•  Look  forward/look  back  feature 

•  Direct  to  feature 

•  Navigation  data 
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3  Downs: 

•  Its  small 

•  It’s  too  hot 

•  Direction  of  travel  for  the  UAVs  was  unclear 

•  Text  format 

•  Lack  of  map  scales 

•  Ease  of  use  of  zoom  feature 

•  Slow  to  load  map  data 

1 1.  In  what  ways  did  the  digital  note  pad  on  the  EDM  assist  or  hinder  you? 

•  It  was  worthless 

•  Did  not  use 

12.  Compared  to  a  paper  and  pencil  pad,  describe  the  benefits  and  shortcomings  of  the  EDM  note 
pad. 

•  No  benefits 

•  Not  readable 

13.  What  additional  map  navigation  features  beyond  “look  ahead”  “look  back”  and  “zoom” 
would  you  add  to  the  EDM? 

•  Slew  (drag  with  cursor) 

•  Drop  down  boxes 

•  Better  touch  response  and  highlight  the  touched/active  box 

•  Pan  feature  and  the  ability  to  mark  certain  points 

14.  How  did  the  map  navigation  features  support  your  decisions  and  actions  regarding  the 
dynamic  ACM  symbols? 

•  Allowed  navigation  around  the  ACMs 

•  Allowed  me  to  navigate  around  ACMs 
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Appendix  F.  Pilot  Open-Ended  Survey  Responses 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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VIGNETTE  1 

Workload  Comments 

1.  If  you  gave  a  workload  rating  of  ‘6’  or  higher  for  any  task,  explain  why  the  workload  was  high 
for  that  task: 

•  No  Responses 

2.  In  the  mission  you  just  flew,  list  any  flight  and/or  mission  tasks  that  you  had  to  ask  your 
crewmember  to  accomplish  because  you  workload  was  too  high: 

•  No  Responses 

Situational  Awareness  Comments 

1.  Describe  any  instances  when  you  feel  you  had  low  situational  awareness  during  the  mission: 

•  Not  able  to  quickly  plot  received  tactical  SA  on  EDM  due  to  training  on  equipment. 

•  When  the  GPS  data  was  lagging 

EDM  Related  Comments 

1 .  Did  the  copilot  have  adequate  information  to  aid  navigation? 

Yes,  please  explain 

•  Route  on  EDM  was  sufficient  to  navigate  aircraft 
No,  please  explain 

•  No,  with  the  GPS  data  out,  we  got  off  course 

2.  How  did  the  EDM  support  the  copilot’s  need  for  navigational  infonnation,  from  your 
perspective? 

•  What  was  loaded  by  AMPS  was  sufficient  until  new  ACMs  were  received.  Any  new 
information  would  have  to  be  manually  looked  up  to  confirm  if  it  interfered  with  the 
route  of  flight  or  mission 

•  Didn’t  give  good  SA.  Didn’t  know  where  some  of  the  SA  given  was  located,  and 
couldn’t  plot  it. 

3.  Explain  how  radio  transmitted  information  helped  you  in  your  mission 

•  Gave  us  a  heads  up  to  the  possibility  of  ACMs  that  might  affect  the  mission 

•  Let  us  know  to  look,  but  we  didn’t  know  where  to  look 

4.  Please  provide  any  other  comments  regarding  how  the  EDM  supported  your  mission? 

•  We  were  able  to  use  it  for  navigation  only 

5.  Please  name  3  likes  and  3  dislikes  regarding  your  experience  with  the  EDM? 

3  Ups: 

•  Moving  maps 

•  Route  information 

•  Navigation  information 
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3  Downs: 

•  Lack  of  EDM  training 


VIGNETTE  2 

Workload  Comments 

1 .  If  you  gave  a  workload  rating  of  ‘6’  or  higher  for  any  task,  explain  why  the  workload  was  high 
for  that  task: 

•  No  Responses 

2.  In  the  mission  you  just  flew,  list  any  flight  and/or  mission  tasks  that  you  had  to  ask  your 
crewmember  to  accomplish  because  you  workload  was  too  high: 

•  No  Responses 

Situational  Awareness  Comments 

1.  Describe  any  instances  when  you  feel  you  had  low  situational  awareness  during  the  mission: 

•  None  as  pilot  on  controls 

•  During  loss  of  GPS  data,  all  SA  data  became  stale 

EDM  Related  Comments 

1 .  Did  the  copilot  have  adequate  information  to  aid  navigation? 

Yes,  please  explain 

•  Route  information  provided  sufficient  navigation  guidance  to  perform  mission 

•  Yes,  a  mix  of  EDM  data  and  dead  reckoning 

•  Yes,  same  as  before 
No,  please  explain 

•  No,  with  the  GPS  data  out,  we  got  off  course 

2.  How  did  the  EDM  support  the  copilot’s  need  for  navigational  information,  from  your 
perspective? 

•  Provided  basic  navigation  information 

•  He  was  able  to  cross  reference  the  EDM  and  track  data 

•  Same  as  before 

3.  Please  provide  any  other  comments  regarding  how  the  EDM  supported  your  mission? 

•  Having  ACMs  appear  allowed  to  the  crew  to  develop  alternate  COAs  during  the  mission 

•  Good  SA  to  the  flying  pilot 

4.  Please  name  3  likes  and  3  dislikes  regarding  your  experience  with  the  EDM? 

3  Ups: 

•  Navigation  data  provided 

•  ACM  updates  were  great 

•  Look  ahead  feature 
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•  It  aided  navigation 

•  Great  visual  cueing  to  flight  hazards 
3  Downs: 

•  Notepad  was  unreadable 

•  Maps  were  slow  to  load 

•  Updates  were  slow  to  load 

•  Need  to  expand  look  ahead  functions 

•  Increase  the  processing  speed 


VIGNETTE  3 

Workload  Comments 

1.  If  you  gave  a  workload  rating  of  ‘6’  or  higher  for  any  task,  explain  why  the  workload  was  high 
for  that  task: 

•  No  Responses 

2.  In  the  mission  you  just  flew,  list  any  flight  and/or  mission  tasks  that  you  had  to  ask  your 
crewmember  to  accomplish  because  you  workload  was  too  high: 

•  No  Responses 

Situational  Awareness  Comments 

1.  Describe  any  instances  when  you  feel  you  had  low  situational  awareness  during  the  mission: 

•  Any  time  the  GPS  sync  was  lost,  specifically  when  nearing  a  turn  or  ROZ. 

EDM  Related  Comments 

1 .  Did  the  copilot  have  adequate  information  to  aid  navigation? 

Yes,  please  explain 

•  Yes,  the  information  presented  was  adequate.  When  able  to  view  the  course  data  it 
allowed  for  timely  course  corrections 

•  Navigation  data  provided  guidance  for  aircraft 

•  All  SA  was  adequate 
No,  please  explain 

•  No  responses 

2.  How  did  the  EDM  support  the  copilot’s  need  for  navigational  infonnation,  from  your 
perspective? 

•  Visual  cueing  and  guidance.  Also,  graphical  representations  of  hazards 

•  Provided  all  necessary  information  to  conduct  mission 

•  SA  allowed  us  to  reroute  when  needed 

3.  Explain  how  radio  transmitted  information  helped  you  in  your  mission. 

•  It  brought  attention  to  the  hazard.  It  was  an  initial  “yeah  you,  be  ready  for  map  data 
update” 
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•  Gave  heads  up  to  mission  changes  that  might  affect  route  of  flight 

4.  Please  provide  any  other  comments  regarding  how  the  EDM  supported  your  mission? 

•  Excellent  mission  aid,  but  needs  a  little  more  development 

•  Provided  SA  to  make  decisions  that  affected  mission 

•  Allowed  us  to  maintain  airspace  surveillance 

5.  Please  name  3  likes  and  3  dislikes  regarding  your  experience  with  the  EDM? 

3  Ups: 

•  Route  data 

•  Flight  hazard  data 

•  Real  time  updates 

•  Route  information 

•  Look  ahead/behind/zoom  function 
3  Downs: 

•  GPS  sync  loss 

•  Text  data  positioning  and  size 

•  ROZ  box  translucence  was  not  good 

•  Lack  of  map  scales  available 

•  All  text  was  not  oriented  in  relation  to  route  of  flight 

•  Fonts  too  small  for  cross  cockpit  viewing 
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1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  ARMC  FIELD  ELEMENT 

RDRL  HRM  CH  C  BURNS 
THIRD  AVE  BLDG  1467B  RM  336 
FORT  KNOX  KY  40121 


NO.  OF 

COPIES  ORGANIZATION 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  AWC  FIELD  ELEMENT 

RDRL  HRM  DJ  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FORT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  CK  J  REINHART 
10125  KINGMAN  RD  BLDG  317 
FORT  BELVOIR  VA  22060-5828 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  AY  M  BARNES 
2520  HEALY  AVE 
STE  1172  BLDG  51005 
FORT  HUACHUCA  AZ  85613-7069 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  AP  D  UNGV ARSKY 
POPE  HALL  BLDG  470 
BCBL  806  HARRISON  DR 
FORT  LEAVENWORTH  KS  66027-2302 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  AT  J  CHEN 
12423  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  AT  C  KORTENHAUS 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 
(HC)  RDRL  HRM  CU  B  LUT  AS  -SPENCER 
6501  E  11  MILE  RD  MS  284 
BLDG200A  2ND  FL  RM2104 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  FIRES  CTR  OF  EXCELLENCE 
FIELD  ELEMENT 
RDRL  HRM  AF  C  HERNANDEZ 
3040  NW  AUSTIN  RD  RM  221 
FORT  SILL  OK  73503-9043 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  AV  W  CULBERTSON 
91012  STATION  AVE 
FORT  HOOD  TX  76544-5073 
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NO.  OF 

COPIES  ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


RDRL  HRS 
J  LOCKETT  (PDF) 
RDRL  HRS  B 
M  LAFIANDRA  (PDF) 
RDRL  HRS  C 
K  MCDOWELL  (PDF) 
RDRL  HRS  D 
B  AMREIN  (PDF) 
RDRL  HRS  E 
D  HEADLEY  (PDF) 


8  ARMY  RSCH  LABORATORY  -  HRED 
(5  PDF,  SIMULATION  &  TRAINING 
3  HC)  TECHNOLOGY  CENTER 

RDRLHRT  COL  M  CLARKE  (HC) 
RDRLHRT  I  MARTINEZ  (PDF) 
RDRLHRT  R  SOTTILARE  (HC) 
RDRL  HRT  B  N  FINKELSTEIN  (HC) 
RDRL  HRT  G  A  RODRIGUEZ  (PDF) 
RDRL  HRT  I  J  HART  (PDF) 

RDRL  HRT  M  C  METEVIER  (PDF) 
RDRL  HRT  S  B  PETTIT  (PDF) 

12423  RESEARCH  PARKWAY 
ORLANDO  FL  32826 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  (PDF)  HQ  USASOC 

RDRL  HRM  CN  R  SPENCER 
BLDG  E2929  DESERT  STORM  DRIVE 
FORT  BRAGG  NC  28310 

1  ARMY  G1 
(PDF)  DAPE  MR  B  KNAPP 

300  ARMY  PENTAGON  RM  2C489 
WASHINGTON  DC  20310-0300 


1  ARMY  RSCH  LABORATORY  -  HRED 
(HC)  HUMAN  RSRCH  AND  ENGRNG 

DIRCTRT  MCOE  FIELD  ELEMENT 
RDRL  HRM  DW  C  CARSTENS 
6450  WAY  ST 
BLDG  2839  RM310 
FORT  BENNING  GA  31905-5400 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  A  MARES 

1733  PLEASONTON  ROAD,  BOX  3 
FORT  BLISS  TX  79916-6816 


ABERDEEN  PROVING  GROUND 


12  DIR  USARL 
(11  PDF,  RDRL  HR 
1  HC)  L  ALLENDER  (PDF) 

P  FRANASZCZUK  (PDF) 

C  COSENZO  (PDF) 

RDRL  HRM 

P  S  A V AGE-KNEPS  HIELD  (PDF) 
RDRL  HRM 
C  PAULILLO  (HC) 

RDRL  HRM  B 
C  SAMMS  (PDF) 

RDRL  HRM  C 
L  GARRETT  (PDF) 
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Intentionally  lelt  blank. 
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